Compilation and Communication Strategies for Out-of-Core Programs on Distributed Memory Machines

It is widely acknowledged that improving parallel I/O performance is critical for widespread adoption of high performance computing. In this paper, we show that communication in out-of-core distributed memory problems may require both inter-processor communication and le I/O. Thus, in order to improve I/O performance, it is necessary to minimize the I/O costs associated with a communication step. We present three methods for performing communication in out-of-core distributed memory problems. The rst method called the generalized collective communication method follows a loosely synchronous model; computation and communication phases are clearly separated, and communication requires permutation of data in les. The second method called the receiver-driven in-core communication considers only communication required of each in-core data slab individually. The third method called the ownerdriven in-core communication goes even one step further and tries to identify the potential future use of data (by the recipients) while it is in the sender's memory. We describe these methods in detail and present a simple heuristic to choose a communication method from among the three methods. We then provide performance results for two out-of-core applications, the two-dimensional FFT code and the twodimensional elliptic Jacobi solver. Finally, we discuss how the out-of-core and in-core communication methods can be used in virtual memory environments on distributed memory machines. Compilation and Communication Strategies for Out-of-core programs on Distributed Memory Machines Rajesh Bordawekar Alok Choudhary Electrical and Computer Engineering Department 121, Link Hall, Syracuse University, Syracuse, NY 13244 rajesh, [email protected] URL: http://www.cat.syr.edu/~frajesh,choudharg J. Ramanujam ECE Dept., Louisiana State University, Baton Rouge, LA 70803 [email protected] URL: http://www.ee.lsu.edu/jxr/jxr.html